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GRAPHICAL USER INTERFACES HAVING SOUND 
EFFECTS FOR OPERATING CONTROL ELEMENTS AND 
DRAGGING OBJECTS 

5 RELATED APPLICATIONS 

This application is related to U.S. Patent Application Serial No. 08/243,327 
entitled "A System and Method for Customizing Appearance and Behavior of Graphical 
User Interfaces" and to U.S. Patent Application Serial No. 08/243,268 entitled 
"Switching Between Appearance/Behavior Themes in Graphical User Interfaces," and 

10 to U.S. Patent Application Serial No. 08/242,963 entitled "Pattern and Color 

Abstraction in a Graphical User Interface," each of which was filed on May 16, 1994 
and each of which is incorporated here by reference. This application is also related to 
U.S. Patent Application Serial No. 08/435,387, entitled "Graphical User Interfaces 
Having Animated Control Elements" , filed on May 5, 1995, the disclosure of which is 

15 incorporated here by reference. 

COPYRIGHT NOTICE 

A portion of the disclosure of this patent document contains material which is 
subject to copyright protection. The copyright owner has no objection to the facsimile 
20 reproduction of the patent disclosure as it appears in the Patent and Trademark Office 
patent file or records, but otherwise reserves all copyright rights whatsoever. 

BACKGROUND 

The present invention relates generally to graphical user interfaces for computer 
25 systems. More particularly, the present invention relates to systems and methods for 
interfacing applications and operating systems which provide for added behavior, in the 
form of sound effects, over object operation and movement in graphical user interfaces. 

The evolution of the computer industry is unparalleled in its rate of growth and 
complexity. Personal computers, for example, which began as little more than feeble 
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calculators with limited memory, tape-driven input and monochrome displays are now 
able to tackle almost any data processing task. While this meteoric increase in power 
was almost sufficient to satisfy the demand of application programmers and end users 
alike, the corresponding increase in complexity created an ease-of-use problem which 
5 the industry was somewhat slower in solving. Thus, designers were faced with a new 
challenge: to harness this computing power in a form usable by even those with 
relatively little computer training to smooth the transition of other industries into a 
i< computer-based information paradigm. 

As a result, in the early to mid-1980 ? s many new I/O philosophies, such as "user 
W 10 friendly," "WYSIWYG" and "menu driven" came to the forefront of the industry, 
01 These concepts are particularly applicable to microcomputers, also known as personal 
computers, which are intended to appeal to a broad audience of computer users, 
including those who previously feared and mistrusted computers. An important aspect 
of computers which employ these concepts was, and continues to be, the interface 
t p 15 which allows the user to input commands and data and receive results, which is 
JlT commonly referred to as a graphical user interface (GUI). 

One type of GUI display is based on a visual metaphor which uses a monitor 
screen as a work surface called a "desktop" where documents are presented in 
relocatable regions termed "windows". The user interacts with the computer by, for 
20 example, moving objects on the desktop, choosing commands from menus, and 
manipulating window control elements, such as checkboxes and scroll bars. An 
exemplary desktop screen is reproduced as Figure 1. Therein, various GUI objects 
including windows and icons are depicted. 

Windows are one example of GUI objects which can be virtually any size, 
25 shape, or color. Some standard types of windows are commonly predefined for the 
interface including, for example, a document window and a dialog box. One example 
of a standard for a document window is illustrated in Figure 2. Each document window 
which conforms to this standard has a title bar with a title drawn in a system-defined 
font and color. Active document windows can also have control elements as illustrated 
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in Figure 2, for example, a close box, a zoom box, a size box, and scroll bars. Other 
control elements, not shown in Figure 2, include buttons, menu items and checkboxes. 
Those skilled in the art will recognize that myriad other control elements are known and 
in use. 

5 Control elements are on-screen objects that a user can manipulate with, for 

example, a mouse to cause an immediate action or modify settings to cause a future 
action. For example, a button control element can provided for the graphical user 
interface to receive user input as to a selected function to be performed by the system. 
Figure 3 illustrates a window entitled "Find" having three button control elements 

10 whose titles describe the function to be performed when these control elements are 
activated. A user can activate a button control element in a variety of ways, for 
example by positioning a cursor (not shown in Figure 3) over the control element and 
providing an activating input, e.g., depressing a mouse button. 

As shown in Figure 3, control elements are typically displayed in a static default 

15 or normal state. However, when a button control element is actuated the graphical user 
interface redraws the control element in a second static state which visually informs the 
user that the system has recognized the user's input and is performing the desired 
function. For example, some conventional graphical user interfaces redraw button 
control elements in a depressed state to indicate activation. However, most software 

20 applications provide little or no corresponding sound behavior to accompany the 
transition between states. 

The success of these types of conventional interfaces is evident from the number 
of companies which have emulated the desktop environment. Even successful concepts, 
however, must continually be improved in order to keep pace with the rapid growth in 

25 this industry. The advent of multimedia, especially CD-ROM devices, has provided 
vast quantities of secondary storage which have been used to provide video capabilities, 
e.g., live animation and video clips, as regular components of applications. With these 
new resources at their disposal, application designers, and others, desire additional 
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control over the appearance and behavior of the display, including the desktop 
environment and, in particular, objects and control elements on the desktop. 

In the related application entitled "Graphical User Interfaces Having Animated 
Control Elements", it has been described how to provide additional control over the 
5 appearance of, for example, control elements by providing animated transition effects. 
The present invention expands upon these concepts by describing various techniques for 
creating realistic sound effects which can be "provided to support object actions and 
enhance the behavior of the graphical user interface. 

10 SUMMARY 

According to exemplary embodiments of the present invention, improved 
behavioral characteristics can be provided to GUIs by providing sound effects 
associated with object operations. For example, one-shot sounds can be output when 
control elements are operated, e.g., a sound associated with actuating a button control 

15 element. The volume, pitch, delay and frequency of the sound effect can be varied 
either randomly or by the theme designer. 

According to other exemplary embodiments of the present invention, a more 
complicated sound effect can be provided for GUI operations which involve dragging an 
object, e.g., an icon or a window, across the display. In these embodiments, a portion 

20 of the sound effect is repeated to provide for a variable length sound effect determined 
by the length of the drag operation. Panning can be incorporated by mixing stereo 
channels to provide an auditory effect of movement. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 The foregoing, and other, objects, features and advantages of the present 

invention will be more readily understood by those skilled in the art upon reading the 
following detailed description in conjunction with the drawings in which: 

Figure 1 shows a conventional desktop screen; 

Figure 2 shows a conventional document window; 
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Figure 3 illustrates an exemplary window having button control elements; 

Figure 4A illustrates an exemplary computer system which can be used in 
conjunction with the present invention; 

Figure 4B is a block diagram illustrating various exemplary functional units of 
5 the computer system of Figure 4B; 

Figure 5 is a state diagram used to illustrate sound effect transitions of a control 
element according to an exemplary embodiment of the present invention; 

Figure 6 is an exemplary state matrix used to describe control element 
transitional sound effects according to an exemplary embodiment of the present 
10 invention; 

Figure 6A depicts a sequence of sound effect reproduction involving an 
exemplary one-shot sound effect and looping sound effect; 

Figure 7 illustrates a drag operation wherein a looping sound effect can be 

output; 

15 Figure 8 is a graph illustrating a panning function used to vary sound effect 

volume between speakers according to an exemplary embodiment of the present 
invention; and 

Figure 9 is a graph illustrating random frequency variation determination for 
sound effects according to exemplary embodiments of the present invention. 

20 

DETAILED DESCRIPTION 

The present invention is described herein by way of exemplary, illustrative 
embodiments, some of which use the Macintosh® computer system and visual displays 
created thereby as a reference for explaining the present invention. Those desiring 
25 additional information with respect to the Macintosh® system are referred to Inside 
Macintosh . Vols. 1-6. Of course, those skilled in the art will readily appreciate that 
systems and methods according to the present invention can be applied to any type of 
display system having a user interface. Further, although certain operations associated 
with exemplary GUI objects used in an operating system (as opposed to applications 
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running under the control of the operating system) are used to describe concepts 
according to the present invention, it will be understood that the sound effects described 
by the present invention are applicable to any type of manipulation of any type of 
interface object. 

5 Graphical user interfaces according to the present invention can, for example, be 

used in conjunction with a computer such as that illustrated in Figure 4A and in 
particular with operating systems used to coordinate the operation of such computers. 
Therein, a personal computer is illustrated as having a box 16 which contains, among 
other elements, circuit boards associated with the computer's functions including 

10 memory devices and one or more processors, a keyboard 18, a pointing device 20 and a 
monitor 22 having a display space 24. The computer will also preferably include at 
least two speakers 26 and 28 associated with at least two audio channels driven by an 
audio processing component (not shown) in box 16. More than two speakers and two 
audio channels, e.g., four, may also provided and those skilled in the art will readily 

15 appreciate that the sound effect concepts described below are readily extensible to more 
than two audio channels. Moreover, the speakers may alternatively be built into the 
box 16. 

Of course the personal computer illustrated in Figure 4A is purely exemplary of 
one type of computer system in which graphical user interfaces according to the present 

20 invention may be implemented. For example, although the display space 24 of the 

system depicted in Figure 4A consists of that provided by monitor 22, those skilled in 
the art will recognize that additional display devices could be arranged proximate 
monitor 22 to create a display space bridging several monitors. The circuit boards 
included in box 16 can include boards which accommodate functional units such as 

25 those illustrated by the block diagram of Figure 4B. Therein RAM 300, ROM 320, 
processor 340, and I/O 360 (including sound/audio processing functionality) are all 
interconnected via system bus 380. 

The computer system can also have known I/O devices (e.g., CD drives, floppy 
disk drives, hard drives, etc.) which can store and read programs and data structures 
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used to implement the below-described techniques. These programs and data structures 
can be encoded on such computer-readable media. For example, the theme data files 
including sound effect information described below can be stored on computer-readable 
media and read whenever a theme is selected for GUI operation. 
5 According to exemplary embodiments of the present invention, sound effects 

associated with the operation of GUI objects can be provided. These sound effects can 
be specified for each theme, e.g., as part of the data structure found in a theme data 
file, in a manner that is consistent with the appearance characteristics of that theme, 
e.g., the object geometries and fill patterns. A discussion of themes and methods for 

10 handling themes is unnecessary for understanding the present invention, other than a 
general recognition that a theme represents a predetermined definition of appearance 
and behavior attributes associated with GUI objects and that multiple themes having 
different definitions can be provided to a system wherein the underlying functions of the 
GUI objects remains the same. However, the interested reader is referred to the above- 

15 identified applications or to U.S. Patent Application Serial No. 

entitled "A Multiple Theme Engine Graphical User Interface Architecture", filed on the 
same date as this application, the disclosure of which is incorporated here by reference. 

Sound effects according to the present invention can be categorized as either 
one-shot sounds or looped sounds. One-shot sound effects can be associated with state 

20 changes in objects. State changes occur when an object is operated on, either by a user 
or by an application, to change its state, e.g., checking a checkbox, depressing a 
button, opening and closing menus, opening and closing windows, emptying a trash 
container, mounting a volume, etc. State tables can be used to store information 
identifying the transitional sound effects between states that are associated with 

25 particular operations on an object. An exemplary state diagram table is shown as 

Figure 5 which provides an example of possible states and common state transitions for 
a checkbox control element of a window. This state diagram can then be reduced to a 
state table as shown in Figure 6. 
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As seen in Figures 5 and 6, this exemplary checkbox control element has nine 
possible states Q1-Q9 which can be displayed on the GUI. These states include three 
highlighted display states for each of the control element's three values. For example, 
when a user clicks on an unchecked checkbox (state Ql) using a mouse button, this 
5 action moves the control to its pressed state (state Q4). After the mouse button is 
released, the control returns back to its original state (state Ql) and the application is 
notified of the button which has been pressed. The application then switches the value 
q of the control to its new value, which might be checked (state Q2). 
S Figure 6 is a matrix of all of the possible transitions between states Ql and Q9. 

Y: 10 Transition sound effects are indicated for some of these transitions by the notations Ti- 
ff ] T18 found in some of the matrix locations. For example, transition effect Tl occurs 
O when an empty checkbox (state Ql) becomes checked (state Q2). This effect could be, 
13 for example, a one-shot clicking noise that has been predefined by the theme file for 

\*2 this particular transition output through speakers 26 and 28. Other themes may use 
s F 15 other sound effects, or no sound effect, for the same transition Tl. Note that not all of 
M» the state transitions are required to have transition effects as underscored by the empty 
matrix locations in Figure 6. 

In addition to specifying a particular one-shot sound to be output as a sound 
effect for a particular transition, the present invention also provides the theme designer 
20 with techniques for re-using or adjusting the recorded sound. For example, the data 
structure which defines the sound effect for a particular state transition may include 
variables associated with the gain (volume), delay (period beginning when a state 
change occurs and ending when the output of the sound effect begins) and pitch to be 
used when outputting the identified sound for a particular object transition. Thus, the 
25 same recorded sound effect could be re-used for different state transitions by varying 

one or more of these output parameters. This technique also permits the theme designer 
to make small adjustments in a recorded sound effect without actually re-recording the 
effect. 
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One-shot sound effects associated with state changes are played asynchronously 
until complete, e.g., depressing a button may result in a one-shot sound effect that ends 
when the button is displayed in its fully sunk position. By way of contrast, movement 
of objects across the desktop can be accompanied by a looping sound effect designed to 
5 promote auditory perception of movement. These sound effects are "looping" in the 
sense that certain sound segments are repeated to accommodate the variable length 
nature of these effects. 

Looping sound effects according to the present invention will be described 
herein by way of examples which involve dragging objects across the GUI, e.g., 

10 dragging a scroll bar indicator, a slider, a window or an icon, although those skilled in 
the art will appreciate that looping sound effects can be used for other object operations, 
e.g., resizing of objects. Consider the exemplary folder in Figure 7 being dragged 
from the position indicated by the solid outline of the folder to the position indicated by 
the dotted outline. When a user initiates the drag operation, e.g., by holding down a 

15 mouse button while the cursor is positioned over the folder and beginning to move the 
mouse, the drag sound (for the currently selected theme) will begin and will continue 
until the drag operation is completed. Thus, unlike the one-shot sound effects 
associated with state changes which each have a fixed duration, drag sound effects have 
a variable duration dependent upon the length of the object movement period. This is 

20 accomplished by repeating, or looping, certain sound segments associated with this 
sound effect. 

Exemplary embodiments of the present invention divide drag sounds into three 
segments: an attack segment, a sustain segment and a decay segment. Each of these 
segments can be recorded separately and identified individually in the loaded theme data 
25 file or in the code which runs the theme. The attack segment is played first to initiate 
the drag operation, e.g., when the user begins to move folder A. Then, the sustain 
segment is repeated while the drag operation continues, e.g,. while the folder is moving 
across the screen from its original position to its new position. Once the drag operation 
terminates, the decay segment is played to gradually transition out the sound effect 
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associated with this drag operation, e.g, when the user releases the mouse button after 
the folder A is in the position indicated by the dotted outline in Figure 7, As with the 
one-shot sound effects, the looping sound effects can also be adjusted by gain, delay 
and pitch variables which can be individually established for each effect. 
5 Although described separately here, those skilled in the art will appreciate that 

one-shot sounds can be reproduced while a looping sound effect is being played. For 
example, if a document icon is dragged across the screen,, a corresponding looping 
B sound effect will be played. If the document icon is dropped onto a container, such as a 
m folder or a trash can, the state change associated with the placement of the document 

10 into the container may trigger a one-shot sound which may play at the same time as, for 
J|] example, the decay segment of the looping sound effect associated with the drag 
P operation. These, and other, exemplary coordinations between one-shot and looping 
p sound effects are illustrated in Figure 6 A. 

!*; To further enhance the user's perception of the position of an object on the 

;i;;15 screen or that the object is moving, the one-shot sound effects can be output with a 
(4 volume variation between speakers 26 and 28, and the looping sound effect can be 

actively panned between speakers 26 and 28. This is accomplished by stereo mixing of 
the sound effect between the speakers based on the object's position on the screen to 
achieve an auditory effect which implies location or movement. For the one-shot sound 
20 effect, this simply involves playing the sound effect at a fixed, but different volume 
from each speaker (unless the object that is changing states is in the middle of the 
screen). For looping sound effects, active panning changes the volume of the sound 
effect in each channel based on the cursor position. In the example of Figure 7, the 
sound effect would begin, i.e., with the attack segment, being more loudly output from 
25 speaker 26 than speaker 28 (Figure 4A). As the folder is dragged across the screen the 
loudness ratio between speaker 26 and speaker 28 would gradually shift to being 
roughly equal near the middle of the screen and, ultimately, to being output more 
loudly from speaker 28 as the folder neared the position indicated by the dotted outline. 
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Applicants have discovered that the realism of the panning effect can be 
optimized by mixing the sound effect between the two channels as illustrated in Figure 
8. Therein, the panning relationship is plotted as a function of sound effect volume 
(without taking into account any modifications made to the volume by virtue of the gain 

5 parameter specified in the data structure for this sound effect) versus screen position of 
the dragged object. It will be seen that the volume will vary between a minimum of 
about 50% of its recorded value to a maximum of about 150% of its recorded value 
(excluding gain variations). If a gain value is specified for the sound effect for a 
particular theme, then the gain value is multiplied by the illustrated percentage of 

10 recorded volume, for each channel, to determine an appropriate output volume for the 
object at its current position. As mentioned above, this aspect of the present invention 
can readily be extended to provide for active panning in more than two directions, e.g. 
four, by providing four speakers and audio channels. 

According to other exemplary embodiments of the present invention, frequency 

15 variation can be introduced to provide some auditory variety in both the one-shot and 
looping sound effects. For example, each time that a sound effect is called in response 
to a state change or object movement, the corresponding sound effect can be played at a 
frequency that is pseudo-randomly selected. The frequency for playback can be 
selected from within a frequency envelope centered at the originally recorded frequency 

20 and having, for example, a width of plus or minus about 2.5 percent. By changing the 
frequency slightly from repetition to repetition, exemplary embodiments of the present 
invention avoid user fatigue with respect to the sound effects and promote enablement 
of this functionality. 

At the same time, however, a certain amount of consistency in the frequency of 

25 each sound effect is desirable to preserve the auditory effect intended by the theme 
designer and to allow the user to easily associate the sound effect with the 
corresponding state change or object movement. Thus, according to one exemplary 
embodiment of the present invention, selection of a frequency from within the envelope 
is weighted such that frequencies closer to the originally recorded frequency are more 
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likely to be selected than frequencies toward the edges of the envelope. For example, a 
suitable weighting function is illustrated in Figure 9. Therein, a random number is 
generated within an interval between plus and minus one and each value has a 
corresponding frequency variation associated therewith. It can be seen in the Figure 
5 that the curve is weighted such that fully half of the randomly selected frequency 
variations are within one quarter of the total envelope in this exemplary weighting. 

Various controls can be put into place to provide for an orderly presentation of 
sound effects. For example, sound effect commands may be temporarily overridden to 
avoid an avalanche of effects, e.g., when closing or opening many windows at the same 
10 time. Similarly, drag sound effects can be limited to one at a time to provide for a 
consistent auditory experience. 

Sound effects can be classified in one of a plurality of effect levels. In this way, 
a sound control panel can be provided in the GUI which allows a user to select from 
more sound to less sound at which she or he would prefer to operate the interface at any 
15 given time. Alternatively, or in addition thereto, sounds may also be enabled or 

disabled by a user by functional category. For example, sound effects may be enabled 
or disabled for all windows, all menus, all icons of a certain type, etc. 

The above-described exemplary embodiments are intended to be illustrative in 
all respects, rather than restrictive, of the present invention. Thus the present invention 
20 is capable of many variations in detailed implementation that can be derived from the 
description contained herein by a person skilled in the art. All such variations and 
modifications are considered to be within the scope and spirit of the present invention as 
defined by the following claims. 



